A SAS® Macro to Compute Added Predictive Ability of New Markers Predicting a Dichotomous Outcome

Risk prediction is an important field in applied statistics. For example, in clinical research, predicting the development of adverse medical conditions is the object of many studies. However, this concept is multi-disciplinary. Many published models exist to predict dichotomous outcomes, ranging widely from a model to predict winners in the NCAA basketball tournament to a recently published risk model to predict the likelihood of a bleeding complication after percutaneous coronary intervention. It is important to note, however, these models are dynamic, as changes in population and technology lead to discovery of new and better methods to predict outcomes. An important consideration is when to add new markers to existing models. While a significant p-value is an important condition, it does not necessarily imply an improvement in model performance. Traditionally, receiver operating characteristic (ROC) curves and its corresponding area under the curve (AUC) are used to compare models, with a statistical test due to DeLong et al. for two correlated AUCs. However, the clinical relevance of this metric has been questioned by researchers. To address this issue, Pencina and D’Agostino have proposed two statistics to evaluate the significance of novel markers. The Integrated Discrimination Improvement (IDI) measures the new model’s improvement in average sensitivity without sacrificing average specificity. The Net Reclassification Improvement (NRI) measures the correctness of reclassification of subjects based on their predicted probabilities of events using the new model with the option of imposing meaningful risk categories. Currently, only the AUC can be obtained directly from SAS (proc logistic). In contrast, the AUC as well as the NRI and IDI, can both be outputted in R with the ImproveProb function under the Hmisc library (Click Here). In this paper we present a single SAS macro which allows calculation of all three measures with several user-defined specifications.